The invention relates to measuring, maintaining and correcting synchronization between two signals which suffer varying relative delays during transmission and/or storage, and in particular to measuring the relative delay between multiple audio signals and an associated video signal of a television type program which is compressed via MPEG or other compression method for transmission and/or storage.
1. Field of the Invention
The present invention relates to the field of transmitting and storing multiple electronic signals where synchronization of the signals is of concern. When such transmitting and storing are of a nature which makes the corresponding receiving and recovering of said signals subject to timing errors resulting from differing amounts of processing delays the present invention is useful in measuring the relative timing errors or delays between signals with such delay measurement being used as a meter of quality of the transmitting and storing and for maintaining or correction of relative delays between such signals.
2. Description of Related Prior Art
It is known in the television signal transmission field to measure and correct audio to video timing errors by measuring the delay which a video signal experiences and using that measurement to delay a companion audio signal by a corresponding amount.
U.S. Pat. No. 4,313,135 by the present inventor shows to compare relatively undelayed and delayed versions of the same video signal to provide a delay signal responsive to the delay thereof and to couple that delay signal to a variable audio delay to cause the audio delay to delay the companion audio signal by a corresponding amount.
U.S. Pat. Nos. 4,665,431 and 5,675,388 by the present inventor show transmitting an audio signal as part of a video signal so that both the audio and video signals experience the same transmission delays thus maintaining the relative synchronization therebetween.
U.S. Reissue Pat. RE 33,535 corresponding to U.S. Pat. No. 4,703,355 shows in the preferred embodiment to encode in the vertical interval of a video signal, a timing signal derived from an audio signal and transmitting the combined video signal and the audio signal. At the receiving location the timing signal is recovered from the video signal and a new timing signal is generated from the received audio signal. The two timing signals are compared at the receiving location to determine the relative delay between the timing signal recovered from the video and the newly generated timing signal, thus determining the relative delay between the video and audio signals at the receive location. It is also suggested to put a timing signal in the audio signal.
U.S. Pat. No. 5,202,761 by the present inventor shows in the preferred embodiment to encode a pulse in the vertical interval of a video signal before the video signal is delayed. The encoded pulse is recovered from the vertical interval of the delayed video signal. Various methods responsive to the encoded pulse or the timing thereof for the undelayed video and the encoded pulse recovered from the vertical interval of the delayed video are shown which enable the determination of the delay, or the control of a corresponding audio delay.
U.S. Pat. No. 5,530,483 by the present inventor shows determining video delay by sampling an image of the undelayed video and sampling images, including the same image of the delayed version of the video and comparing the samples of the undelayed image to the samples of the delayed images until a match is found indicating that the undelayed image in delayed form is being compared. The time lapse between the sampling of the undelayed image, and the finding of the matching delayed image is used as a measure of video signal delay.
U.S. Pat. No. 5,572,261 by the present inventor shows a method of determining the relative delay between an audio and a video signal by inspecting the video for a speaker""s mouth and determining various mouth patterns of movement which correspond to sounds which are present in the audio signal. The time relationship between a mouth pattern which creates a sound and the occurrence of that sound in the audio is used as a measure of audio to video timing.
U.S. Pat. No. 5,751,368, a CIP of U.S. Pat. No. 5,530,483 shows the use of comparing samples of relatively delayed and undelayed versions of video signal images for determining the delay of multiple signals.
Applicant incorporates all of the above prior art patents herein as fully as if they were set forth in their entirety for the purposes of enabling one of ordinary skill in the art to practice the present invention in so far as the present invention utilizes many elements which are taught therein. In particular, attention is called to RE 33,535 and the teachings of generating a timing signal in response to an audio signal, and the comparison of a recovered timing signal and a newly generated timing signal at the receiving site to determine the relative delay therebetween.
The above cited inventions often prove to be less than complete solutions for modern television systems and others which transmit or store a plurality of signals for various reasons including for example those problems recited below. In particular, the current transmission of MPEG compressed television signals has proven to have particular difficulty in maintaining audio to video synchronization, and the prior art has particular problems in dealing with such.
U.S. Pat. No. 4,313,135 compares relatively undelayed and delayed versions of the same video signal to provide a delay signal. This method requires connection between the undelayed site and the delayed site and is unsuitable for environments where the two sites are some distance apart. For example where television programs are sent from the network in New York to the affiliate station in Los Angeles such system is impractical because it would require the undelayed video to be sent to the delayed video site in Los Angeles without appreciable delay, somewhat of an oxymoron when the problem is that the transmission itself creates the delay which is part of the problem. A problem also occurs with large time delays such as occur with storage such as by recording since by definition the video is to be stored and the undelayed version is not available upon the subsequent playback or recall of the stored video.
U.S. Pat Nos. 4,665,431 and 5,675,388 show transmitting an audio signal as part of a video signal so that both the audio and video signals experience the same transmission delays thus maintaining the relative synchronization therebetween. This method is expensive for multiple audio signals, and the digital version has proven difficult to implement when used in conjunction with video compression such as MPEG.
U.S. Reissue Pat. RE 33,535 corresponding to U.S. Pat. No. 4,703,355 shows in the preferred embodiment to encode a timing signal in the vertical interval of a video signal and transmitting the video signal with the timing signal. Unfortunately many systems strip out and fail to transmit the entire vertical interval of the video signal thus causing the timing signal to be lost. It is suggested to put a timing signal in the audio signal, which is continuous thus reducing the probability of losing the timing signal. Unfortunately it is difficult and expensive to put a timing signal in the audio signal in a manner which ensures that it will be carried with the audio signal, is easy to detect, and is inaudible to the most discerning listener.
U.S. Pat. No. 5,202,761 shows to encode a pulse in the vertical interval of a video signal before the video signal is delayed. This method also suffers when the vertical interval is lost.
U.S. Pat. No. 5,530,483 shows determining video delay by a method which includes sampling an image of the undelayed video. This method also requires the undelayed video, or at least the samples of the undelayed video, be available at the receiving location without significant delay. Like the ""135 patent above this method is unsuitable for long distance transmission or time delays resulting from storage.
U.S. Pat. No. 5,572,261 shows a method of determining the relative delay between an audio and a video signal by inspecting the video for particular sound generating events such as a particular movement of a speaker""s mouth and determining various mouth patterns of movement which correspond to sounds which are present in the audio signal. The time relationship between a video event such as mouth pattern which creates a sound and the occurrence of that sound in the audio is used as a measure of audio to video timing. This method requires a significant amount of audio and video signal processing to operate.
U.S. Pat. No. 5,751,368, a CIP of U.S. Pat. No. 5,530,483 shows the use of comparing samples of relatively delayed and undelayed versions of video signal images for determining the delay of multiple signals. Like the""483 patent the""368 patent needs for the undelayed video or at least samples thereof to be present at the receiving location.
Attempts have been made to add various timing related signals in television program streams in order to maintain audio to video synchronization. In particular in MPEG systems control signals such as time stamps are utilized. Unfortunately the inclusion of these signals does not guarantee proper audio to video synchronization at the receive side output of the system for a variety of reasons, including the fact that there are significant video delays which occur which can not be tracked by the time stamps.
It is an object of the invention to provide a method for measuring or maintaining the relative delay of a plurality of signals which are passed through subsequent processing.
It is another object of the invention to provide a method of generating a marker in response to a second signal which marker may be associated with a first signal in a fashion that said marker is carried with said first signal through processing of said first signal.
It is still another object of the invention to provide a method of responding to a marker which has been associated with a first signal and a marker which is provided in response to a second signal whereby said markers may be utilized to determine the relative delay between said first and second signals.
It is a further object of the invention to provide a marker in response to a signal wherein said marker indicates the occurrence of particular characteristics of said signal.
It is a still further object of the invention to provide a system of measuring the relative delay between an audio and a video signal in a television system wherein the audio and video signals are subject to differing processing which creates unequal delays in said signals.
It is yet still a further object of the invention to provide a method of marking a first signal which may be a video signal to allow relative delay measurement of said first signal and a second signal which may be an audio signal after they have been processed, including use of a marker generator responsive to the second signal to generate a marker upon the occurrence of one or more particular characteristics of the audio, associating the marker with the video signal in a fashion such that the marker will by carried with the video signal and not be adversely affected by the subsequent processing thereof.
It is yet still another object of the invention to provide a relative delay measurement system for measuring the relative delay between a plurality of signals including a first signal which is a video signal and second signal which is an audio signal which signals experience unequal delays due to processing thereof, the invention including use of a marker generator responsive to the audio signal to generate a marker upon the occurrence of one or more particular characteristics of the audio, associating the marker with the video signal in a fashion such that the marker will by carried with the video signal but not be adversely affected by the subsequent processing thereof, responding to the marker with the video signal after the processing to generate a first delayed marker; generating a second delayed marker in response to the processed audio signal, comparing the relative timing of the first and second delayed markers to determine the relative timing between the processed audio and processed video signal.
The preferred embodiment of the invention may be used with a television signal. At the transmitting location a marker is generated in response to the audio signal and is associated with the video signal such that the marker is carried with the video signal in a fashion such that it will not be lost or adversely affected by the expected processing of the video signal. The audio signal and the marker associated video signal are stored, transmitted and/or processed and made available at a later time thus becoming delayed video and audio signals. A first delayed marker is recovered from the delayed video signal and a corresponding second delayed marker is generated from the delayed audio signal, with the two delayed markers compared to determine the relative delay therebetween. This relative delay between these markers is responsive to and is a measure of the delay between the delayed video signal and delayed audio signal.
Somewhat simplistically stated, the preferred embodiment of the invention operates by generation of the marker at the transmit section, which may be thought of a marking the video at the time of the occurrence of a known event in the audio signal. The time marker is associated with the video signal such that it is carried in time with the video signal for all of the processing which the video signal is to experience. After the video signal processing and any audio signal processing, the same event in the audio is again marked in time, and the previously marked time (relative to the video) is recovered or flagged in the received video. Since it is known that the audio event and the marking of the video occurred (substantially) simultaneously at the transmit location, the displacement between those events at the receive location is a measure of the audio to video timing error, or the relative delay therebetween.
Generally, the present invention teaches measuring the relative delay between a plurality of signals which have suffered differing delays due to transmission, storage or other processing. The preferred embodiment of the invention includes the use of a marker which is generated in response to a second signal and combined with a first signal in a manner which ensures that the marker will not be lost in the expected processing of the first signal. Subsequently a first delayed marker is generated in response to the marker associated with or recovered from the first signal, and a second delayed marker is generated from the second signal. The first delayed marker and second delayed marker are compared to determine a measure of the relative timing or delay between said first signal and said second signal at said subsequent time.